{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 猫狗大战 - 迁移学习的一些尝试"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 模型的训练方式\n",
    "\n",
    "- 深度学习模型可以划分为 **训练** 和 **预测** 两个阶段。\n",
    "\n",
    "\n",
    "- **训练** 又分为两种策略。一种是白手起家从头搭建模型进行训练，一种是通过预训练模型进行训练。\n",
    "\n",
    "\n",
    "- **预测** 相对简单，直接用已经训练好的模型对数据集进行预测即可。（此时采用上面两种策略训练得到的模型都被视为“已训练模型”）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\textbf{Application}\n",
    "\\begin{cases}\n",
    "    \\textbf{Training}\n",
    "    \\begin{cases}\n",
    "        \\color{Blue}{\\textbf{Training From Scratch}} \\\\[2ex]\n",
    "        \\color{Green}{\\textbf{Using Pre-trained Model}}\n",
    "        \\begin{cases}\n",
    "            ^{\\#} \\textrm{1. Transfer Learning} \\\\[2ex]\n",
    "            ^{\\#} \\textrm{2. Extract Feature Vector (Bottle Neck)} \\\\[2ex]\n",
    "            ^{\\#} \\textrm{3. Fine-tune}\n",
    "        \\end{cases}\n",
    "    \\end{cases}\n",
    "    \\\\[3ex]\n",
    "    \\textbf{Inference}: \\textrm{Using Pre-trained Model}\n",
    "\\end{cases}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 迁移学习的三种训练方式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "$^{\\#} \\textbf{1. Transfer Learning}$：冻结预训练模型的全部卷积层，只训练自己定制的全连接层。\n",
    "\n",
    "\n",
    "$^{\\#} \\textbf{2. Extract Feature Vector}$：先计算预训练模型的卷积层对所有训练和测试数据的特征向量，然后抛开预训练模型，只训练自己定制的简配版全连接网络。\n",
    "\n",
    "\n",
    "$^{\\#} \\textbf{3. Fine-tune}$：冻结预训练模型的部分卷积层（通常是靠近输入的大部分卷积层），训练剩下的卷积层（通常是靠近输出的小部分卷积层）和全连接层。\n",
    "\n",
    "> \"Transfer Learning\" 和 \"Fine-tune\" 其实没有严格的区分，只不过后者更常见于形容迁移学习的**后期微调**中。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 三种训练方式的对比\n",
    "\n",
    "- 第一种和第二种训练得到的模型本质上并没有什么区别，但是第二种的计算复杂度要远远好于第一种。\n",
    "\n",
    "\n",
    "- 第三种是对前两种方法的补充，以进一步提升模型性能。要注意的是，这种方法**并不一定能对模型有所提升**。\n",
    "\n",
    "\n",
    "- 本质上来讲，这三种迁移学习的方式都是为了让预训练模型能够胜任新数据集的识别工作，能够让预训练模型原本的特征提取能力得到充分的释放和利用。但是，在此基础上如果想让模型能够达到更低的Loss，那么光靠迁移学习是不够的，靠的更多的还是**模型的结构以及新数据集的数量**。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 开始实验"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. 数据集预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "import cv2\n",
    "import h5py\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from time import time\n",
    "from datetime import datetime\n",
    "from tqdm import tqdm\n",
    "from utils import get_params_count\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "from keras.applications import inception_v3, xception, resnet50, vgg16, vgg19\n",
    "from keras.applications import InceptionV3, Xception, ResNet50, VGG16, VGG19\n",
    "from keras.layers import Input, Dense, Dropout, Activation, Flatten, Lambda\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint, TensorBoard\n",
    "from keras.models import Model\n",
    "from keras.optimizers import SGD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████| 12500/12500 [00:56<00:00, 220.52it/s]\n",
      "100%|█████████████████████████████████████████████████████| 12500/12500 [00:56<00:00, 221.31it/s]\n",
      "100%|█████████████████████████████████████████████████████| 12500/12500 [00:56<00:00, 222.85it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Data Size = 6.24 GB\n",
      "Testing Data Size = 3.12 GB\n"
     ]
    }
   ],
   "source": [
    "height = 299\n",
    "labels = np.array([0] * 12500 + [1] * 12500)\n",
    "train = np.zeros((25000, height, height, 3), dtype=np.uint8)\n",
    "test = np.zeros((12500, height, height, 3), dtype=np.uint8)\n",
    "\n",
    "for i in tqdm(range(12500)):\n",
    "    img = cv2.imread('./train/cat/%s.jpg' % str(i))\n",
    "    img = cv2.resize(img, (height, height))\n",
    "    train[i] = img[:, :, ::-1]\n",
    "    \n",
    "for i in tqdm(range(12500)):\n",
    "    img = cv2.imread('./train/dog/%s.jpg' % str(i))\n",
    "    img = cv2.resize(img, (height, height))\n",
    "    train[i + 12500] = img[:, :, ::-1]\n",
    "\n",
    "for i in tqdm(range(12500)):\n",
    "    img = cv2.imread('./test/%s.jpg' % str(i + 1))\n",
    "    img = cv2.resize(img, (height, height))\n",
    "    test[i] = img[:, :, ::-1]\n",
    "    \n",
    "print('Training Data Size = %.2f GB' % (sys.getsizeof(train)/1024**3))\n",
    "print('Testing Data Size = %.2f GB' % (sys.getsizeof(test)/1024**3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实验一：冻结全部卷积层 + 训练自己定制的全连接层"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 要点\n",
    "\n",
    "- **使用InceptionV3预训练模型**：该模型当初训练时的输入图像尺寸是299x299，且图像通道顺序为RGB，模型总参数为**两千多万个**。\n",
    "- **预处理**：按照预训练模型原本的预处理方式对数据进行预处理。\n",
    "- **基模型**：导入预训练模型，注意只导入卷积层部分，并锁定全部卷积层参数。\n",
    "- **定制模型**：卷积层之后先接全局平均池化（GAP），再接Dropout，再接分类器，根据分类任务选择输出个数。模型可训练参数只有**两千个**。\n",
    "- **优化器**：采用较小学习率的SGD。\n",
    "- **数据准备**：将训练集划分为训练集和验证集。\n",
    "- **定义回调函数以方便训练**：自动在每代结束保存模型，以val_loss作为监控指标进行早停，训练历史数据同步更新到Tensorboard供可视化。\n",
    "- **需要用非常小的batch_size进行训练**：这样可以让模型更快更好的收敛。\n",
    "- 可以看到，虽然卷积层全都已经锁定，但是由于样本依然需要从模型的输入一直计算到输出，因此训练还是比较耗时的，训练五代需要十几分钟。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trainable: 2049, Non-Trainable: 21802784\n"
     ]
    }
   ],
   "source": [
    "# Preprocess: Standardization\n",
    "x = Input(shape=(height, height, 3))\n",
    "x = Lambda(inception_v3.preprocess_input)(x)\n",
    "\n",
    "# Base Model: Freeze all conv layers\n",
    "base_model = InceptionV3(include_top=False, input_tensor=x, weights='imagenet', pooling='avg')\n",
    "for layer in base_model.layers:\n",
    "    layer.trainable = False\n",
    "\n",
    "# Customized Classifier\n",
    "y = Dropout(0.2)(base_model.output)\n",
    "y = Dense(1, activation='sigmoid', kernel_initializer='he_normal')(y)\n",
    "\n",
    "# Full Model: Pre-train Conv + Customized Classifier\n",
    "model = Model(inputs=base_model.input, outputs=y, name='Transfer_Learning')\n",
    "sgd = SGD(lr=1e-3, decay=1e-6, momentum=0.9, nesterov=True)\n",
    "model.compile(loss='binary_crossentropy', optimizer=sgd, metrics=['accuracy'])\n",
    "print('Trainable: %d, Non-Trainable: %d' % get_params_count(model))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_val, y_train, y_val = train_test_split(train, labels, shuffle=True, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 163s - loss: 0.1887 - acc: 0.9367 - val_loss: 0.0853 - val_acc: 0.9798\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 161s - loss: 0.1220 - acc: 0.9566 - val_loss: 0.0700 - val_acc: 0.9818\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 161s - loss: 0.1084 - acc: 0.9604 - val_loss: 0.0601 - val_acc: 0.9834\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 161s - loss: 0.1086 - acc: 0.9602 - val_loss: 0.0599 - val_acc: 0.9824\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 161s - loss: 0.0991 - acc: 0.9630 - val_loss: 0.0565 - val_acc: 0.9822\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 161s - loss: 0.0984 - acc: 0.9626 - val_loss: 0.0514 - val_acc: 0.9840\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 160s - loss: 0.0925 - acc: 0.9649 - val_loss: 0.0509 - val_acc: 0.9840\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 159s - loss: 0.0976 - acc: 0.9629 - val_loss: 0.0480 - val_acc: 0.9848\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 161s - loss: 0.0959 - acc: 0.9653 - val_loss: 0.0479 - val_acc: 0.9848\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 160s - loss: 0.0896 - acc: 0.9672 - val_loss: 0.0476 - val_acc: 0.9842\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1f183902a20>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prepare Callbacks for Model Checkpoint, Early Stopping and Tensorboard.\n",
    "log_name = '/DogVSCat-EP{epoch:02d}-LOSS{val_loss:.4f}.h5'\n",
    "log_dir = datetime.now().strftime('transfer_model_%Y%m%d_%H%M')\n",
    "if not os.path.exists(log_dir):\n",
    "    os.mkdir(log_dir)\n",
    "\n",
    "es = EarlyStopping(monitor='val_loss', patience=20)\n",
    "mc = ModelCheckpoint(log_dir + log_name, monitor='val_loss', save_best_only=True)\n",
    "tb = TensorBoard(log_dir=log_dir)\n",
    "\n",
    "model.fit(x=X_train, y=y_train, batch_size=16, epochs=10, validation_data=(X_val, y_val), callbacks=[es, mc, tb])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实验二：导出特征向量，再单独训练分类器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 要点\n",
    "\n",
    "- **预处理**：导出训练集和测试集的特征向量之前，一定要记得按照预训练模型的要求进行预处理，否则导出的特征将不是模型的最佳表现。\n",
    "- **基模型**：基模型同样由InceptionV3的卷积层部分以及全局平均池化（GAP）构成。\n",
    "- **导出即预测**：所谓导出，其实就是让基模型直接对训练集和测试集进行预测，只不过预测出的不是类别，而是特征向量（特征图的浓缩版本）。\n",
    "- **导出需要一定时间**：由于导出需要对数据集的所有图片进行预测，因此通常需要一两分钟才能完成。好的是，一旦完成，就可以完全和CNN拜拜了。\n",
    "- **新模型的输入是特征向量**：新模型的输入不再是训练集的图像本身，而是经过预训练模型“消化”后的图像特征向量，该向量的第一个维度对应于每一个图像样本，其长度为样本的个数，第二个维度是基模型最后一层每个卷积核输出特征图的平均值，对于InceptionV3来说，第二个维度的长度是2048。\n",
    "- **划分训练集和验证集**：为了训练时了解模型收敛情况，同样对特征向量划分训练集和验证集。\n",
    "- **定制新模型**：由于已经导出特征向量，因此接下来只需训练一个输入特征长度为2048的全连接网络即可。\n",
    "- 同样采用回调函数以及很小的batch_size（16）进行训练。可以看到训练快到飞起，5代训练仅用时十几秒钟，就可以达到0.02左右的val_loss。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Preprocess: Standardization\n",
    "x = Input(shape=(height, height, 3))\n",
    "x = Lambda(inception_v3.preprocess_input)(x)\n",
    "\n",
    "# Base Model: Extract feature vector of both train & test dataset\n",
    "base_model = InceptionV3(include_top=False, input_tensor=x, weights='imagenet', pooling='avg')\n",
    "train_gap = base_model.predict(train, batch_size=128)\n",
    "test_gap = base_model.predict(test, batch_size=128)\n",
    "\n",
    "X_train_gap, X_val_gap, y_train_gap, y_val_gap = train_test_split(train_gap, labels, shuffle=True, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trainable: 2049, Non-Trainable: 0\n"
     ]
    }
   ],
   "source": [
    "# Input Shape: (Batch Size, Feature Vector length)\n",
    "x = Input(shape=(X_train_gap.shape[1],))\n",
    "y = Dropout(0.2)(x)\n",
    "y = Dense(1, activation='sigmoid', kernel_initializer='he_normal', name='classifier')(y)\n",
    "model_gap = Model(inputs=x, outputs=y, name='GAP')\n",
    "model_gap.compile(loss='binary_crossentropy', optimizer='adadelta', metrics=['accuracy'])\n",
    "print('Trainable: %d, Non-Trainable: %d' % get_params_count(model_gap))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "20000/20000 [==============================] - 3s - loss: 0.0754 - acc: 0.9856 - val_loss: 0.0317 - val_acc: 0.9912\n",
      "Epoch 2/5\n",
      "20000/20000 [==============================] - 3s - loss: 0.0270 - acc: 0.9926 - val_loss: 0.0261 - val_acc: 0.9918\n",
      "Epoch 3/5\n",
      "20000/20000 [==============================] - 3s - loss: 0.0219 - acc: 0.9936 - val_loss: 0.0241 - val_acc: 0.9922\n",
      "Epoch 4/5\n",
      "20000/20000 [==============================] - 3s - loss: 0.0199 - acc: 0.9941 - val_loss: 0.0234 - val_acc: 0.9924\n",
      "Epoch 5/5\n",
      "20000/20000 [==============================] - 3s - loss: 0.0191 - acc: 0.9940 - val_loss: 0.0229 - val_acc: 0.9920\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1f181369e80>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prepare Callbacks for Model Checkpoint, Early Stopping and Tensorboard.\n",
    "log_name = '/DogVSCat-EP{epoch:02d}-LOSS{val_loss:.4f}.h5'\n",
    "log_dir = datetime.now().strftime('gap_model_%Y%m%d_%H%M')\n",
    "if not os.path.exists(log_dir):\n",
    "    os.mkdir(log_dir)\n",
    "\n",
    "es = EarlyStopping(monitor='val_loss', patience=20)\n",
    "mc = ModelCheckpoint(log_dir + log_name, monitor='val_loss', save_best_only=True)\n",
    "tb = TensorBoard(log_dir=log_dir)\n",
    "\n",
    "model_gap.fit(x=X_train_gap, y=y_train_gap, batch_size=16, epochs=5, validation_data=(X_val_gap, y_val_gap), callbacks=[es, mc, tb])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实验三：尝试对模型进行微调，以进一步提升模型性能"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Fine-tune所扮演的角色\n",
    "\n",
    "拿到新数据集，想要用预训练模型处理的时候，通常大家都会先用上面实验一或者实验二里的方法看看预训练模型在新数据上的表现怎么样，摸个底。如果表现不错，还想看看能不能进一步提升，就可以试试Fine-tune（即解锁比较少的卷积层继续训练），但是不要期待会有什么质的飞跃。如果由于新数据集与原数据集（例如ImageNet数据集）的差别太大导致表现很糟，那么一方面可以考虑自己从头训练模型，另一方面也可以考虑解锁比较多层的训练，亦或干脆只用预训练模型的参数作为初始值，对模型进行完整训练。\n",
    "\n",
    "### 2. Fine-tune的三种实现方式\n",
    "\n",
    "其实基本思路都是一样的，就是解锁少数卷积层继续对模型进行训练。\n",
    "\n",
    "- **场景1**：已经采用实验一里的方法，带着冻僵的卷积层训练好分类器了，现在想要进一步提升模型。\n",
    "\n",
    "    - 实现方式：接着用实验一里的模型，直接解锁一小部分卷积层接着训练就成了，很简单。\n",
    "    \n",
    "    \n",
    "- **场景2**：已经采用实验二里的方法，光把分类器训练好了，现在想要进一步提升模型。\n",
    "\n",
    "    - 实现方式：重新搭一个预训练模型接新分类器，然后把实验二里训练好的分类器参数载入到新分类器里，解锁一小部分卷积层接着训练。\n",
    "    \n",
    "    \n",
    "- **场景3**：刚上手，想要 Transfer Learning 和 Fine-tune 一气呵成。（这么做需要搭配很低的学习率，因此收敛可能会很慢）\n",
    "\n",
    "    - 实现方式：和实验一里的操作一样，唯一不同的就是只冻僵一部分卷积层训练。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. 具体实现 \n",
    "\n",
    "下面我们接着实验二的脚步进一步对模型进行Fine-tune（即上面所说的场景2）。\n",
    "\n",
    "#### 要点\n",
    "\n",
    "- **基模型和定制模型**：构建和实验一里面完全相同的模型。要注意的是得把Dense层的名字定义的和实验二中一样（都叫'classifier'）。\n",
    "- **导入分类器的参数**：由于实验二中已经将分类器（即Dense层）训练好了，因此这里只需要by_name的load_weight就可以让刚搭的模型满血复活了。\n",
    "- **通过evaluate函数查看模型状态**：载入权重之前，模型的预测准确率很低，跟瞎猜（50%）差不多。而载入之后，准确率直接飙升到99%，说明权重载入的过程没有问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trainable: 2049, Non-Trainable: 21802784\n",
      "Loss: 0.8281, Acc: 41.70%\n"
     ]
    }
   ],
   "source": [
    "# Preprocess: Standardization\n",
    "x = Input(shape=(height, height, 3))\n",
    "x = Lambda(inception_v3.preprocess_input)(x)\n",
    "\n",
    "# Base Model: Freeze all conv layers\n",
    "base_model = InceptionV3(include_top=False, input_tensor=x, weights='imagenet', pooling='avg')\n",
    "for layer in base_model.layers:\n",
    "    layer.trainable = False\n",
    "\n",
    "# Customized Classifier\n",
    "y = Dropout(0.2)(base_model.output)\n",
    "y = Dense(1, activation='sigmoid', kernel_initializer='he_normal', name='classifier')(y)\n",
    "\n",
    "# Full Model: Pre-train Conv + Customized Classifier\n",
    "model_finetune = Model(inputs=base_model.input, outputs=y, name='Fine-tuning')\n",
    "sgd = SGD(lr=1e-3, decay=1e-6, momentum=0.9, nesterov=True)\n",
    "model_finetune.compile(loss='binary_crossentropy', optimizer=sgd, metrics=['accuracy'])\n",
    "print('Trainable: %d, Non-Trainable: %d' % get_params_count(model_finetune))\n",
    "\n",
    "# Raw Performance (Classifier not trained)\n",
    "res = model_finetune.evaluate(X_train[:1000], y_train[:1000], verbose=0)\n",
    "print(\"Loss: %.4f, Acc: %.2f%%\" % (res[0], res[1] * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss: 0.0142, Acc: 99.40%\n"
     ]
    }
   ],
   "source": [
    "# Load weights from previously trained Dense Layer.\n",
    "weight_path = './gap_model_20171014_1137/DogVSCat-EP04-LOSS0.0229.h5'\n",
    "model_finetune.load_weights(weight_path, by_name=True)\n",
    "\n",
    "# Performance with Trained Classifier \n",
    "res = model_finetune.evaluate(X_train[:1000], y_train[:1000], verbose=0)\n",
    "print(\"Loss: %.4f, Acc: %.2f%%\" % (res[0], res[1] * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![image](https://github.com/mtyylx/Resources/blob/master/CNN%20-%20Model%20InceptionV3.png?raw=true)\n",
    "![image](https://github.com/mtyylx/Resources/blob/master/CNN%20-%20Inception%20Module.png?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. 如何选取要解锁的区域\n",
    "\n",
    "- 如上图所示，由于我们使用的InceptionV3模型，是按照一个一个的Inception Module级联起来的，因此我们要选择Fine-tune的话应该也以Inception Module为最小单位（而不是单独的某一层）进行锁定和解锁。\n",
    "\n",
    "\n",
    "- 通过看 model_finetune.summary() 可以基本确定上图中的 mixed10 区域（倒数34层）构成的 Inception Module 可以解锁训练下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Decide which layers to unlock\n",
    "for layer in model_finetune.layers[-34:]:\n",
    "    layer.trainable = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "20000/20000 [==============================] - 186s - loss: 0.1276 - acc: 0.9520 - val_loss: 0.0650 - val_acc: 0.9798\n",
      "Epoch 2/5\n",
      "20000/20000 [==============================] - 161s - loss: 0.1095 - acc: 0.9587 - val_loss: 0.0637 - val_acc: 0.9812\n",
      "Epoch 3/5\n",
      "20000/20000 [==============================] - 162s - loss: 0.1060 - acc: 0.9587 - val_loss: 0.0546 - val_acc: 0.9830\n",
      "Epoch 4/5\n",
      "20000/20000 [==============================] - 161s - loss: 0.0983 - acc: 0.9623 - val_loss: 0.0566 - val_acc: 0.9824\n",
      "Epoch 5/5\n",
      "20000/20000 [==============================] - 162s - loss: 0.1033 - acc: 0.9611 - val_loss: 0.0522 - val_acc: 0.9830\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x1f8602147f0>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prepare Callbacks for Model Checkpoint, Early Stopping and Tensorboard.\n",
    "log_name = '/DogVSCat-EP{epoch:02d}-LOSS{val_loss:.4f}.h5'\n",
    "log_dir = datetime.now().strftime('finetune_model_%Y%m%d_%H%M')\n",
    "if not os.path.exists(log_dir):\n",
    "    os.mkdir(log_dir)\n",
    "\n",
    "es = EarlyStopping(monitor='val_loss', patience=20)\n",
    "mc = ModelCheckpoint(log_dir + log_name, monitor='val_loss', save_best_only=True)\n",
    "tb = TensorBoard(log_dir=log_dir)\n",
    "\n",
    "model_finetune.fit(x=X_train, y=y_train, batch_size=16, epochs=5, validation_data=(X_val, y_val), callbacks=[es, mc, tb])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 可以看到，fine-tune后模型的Loss反而变差了，从实验二时的0.02上升到了0.05。这说明这个模型的能力也就到此为止了，无法再通过Fine-tune进行提高。如果还想提高的话，只能往两个方向努力：\n",
    "\n",
    "> - 修改模型结构（包括融合更多模型）\n",
    "\n",
    "> - 获得更多数据（包括图像增强等方法）"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
